Live entertainment events such as theatre performances, television and film production, concerts, theme parks, night clubs and sporting events commonly use very large and complex lighting and video arrangements to allow the designers full artistic control over the spectacle being shown to the audience. In order to manage these systems, there has been steady development into highly sophisticated control systems capable of handling thousands of controlled lighting instruments. Examples of lighting instruments include everything from a simple spotlight where the only controllable parameter is the intensity of the luminaire, through fully controllable automated lights where, not only is intensity remotely controllable, but also color, beam shape, movement and position, focus and many other parameters. In recent years we have also seen an explosion in the use of LED based luminaires where arrays of differently colored emitters, perhaps red, green and blue, may be controlled in real time to provide dynamic color effects. In addition, the entertainment technology industry has seen increasing use of video based products such as projectors and LED based video walls where the designer potentially has individual control over every pixel of a display. With a large lighting rig at a concert commonly containing hundreds of lighting instruments as well as myriads of pixel mapped video displays, the need for control systems that reduce the complexity of the system for the operator and provide assistance in managing thousands of control channels in real time has become paramount. FIG. 1 illustrates a typical. lighting control system 10 with a control desk 11 connected via data-links 12 to controlled devices. The controlled devices may include, but not be limited to, automated luminaires 20, non-automated luminaires 21, LED luminaires 22, LED array luminaires 23, video projectors 24, pixel mapped video wall 25, and lasers 26 any similar light emitting and imaging devices.
Historically lighting control systems have been linearly programmed systems, where every parameter of every attached device can be accessed individually or in groups, adjusted, and stored for later retrieval and playback. The operator must work through each and every luminaire or video device they wish to use and set the relevant parameters for every cue. This gives the operator complete control but is very time consuming and, with some of the huge systems in use today, may actually be impossible to achieve within the time constraints of the event. This programming methodology also makes no allowance for changing conditions during live events—the programmed show is frozen and will be played back verbatim unless manually adjusted from the control system by an operator. This is an asset in that the lighting performance will precisely match the pre-programmed rehearsal, but is also a constraint as it does not allow the lighting to follow variations in the performance that are common in live events. There have been many attempts to improve lighting and show control systems to provide the operator with the ability to dynamically modify the live show in real time by means such as manual overrides and the exposing of some parameters as real-time controls. However such systems are still operator constrained and the control system itself provides no direct assistance other than allowing the user to override pre-programmed values. A highly skilled operator familiar with the particular lighting program is always needed and, even then, there are limitations as to what they are physically capable of modifying during a rapidly changing live event.
An example of an early prior art system controller that attempted to address these issues is illustrated in FIG. 2. This lighting control system concept from the early 1990's was aimed at the then burgeoning night club and rave market. The intent was that the lighting controller was not linearly programmed step by step, cue by cue, as described above, but instead just configured by the installer. The lighting looks would then be generated algorithmically by the controller itself at run time in response to a highly abstracted user interface and audio or MIDI input.
This prior art system to control conventional entertainment lighting instruments, automated moving lights in particular. Configuration by the installer entailed selecting the connected luminaires from a library, positioning them in 3D space, and storing within the system some critical positions for the luminaires.
The controller's user interface is shown in FIG. 2. The central principle was based around categorizing lighting looks as levels of “heat” through the grid 15 of Twenty (20) backlit buttons 14 to the left (Marked Red, Amber, Yellow, Olive and Green). The Two (2) rotary knobs 16 and 17 marked Heat set the top and bottom heat levels of the grid's range respectively. In this way, the entire grid 21 could be set to the same temperature, a wide or a narrow range as required to suit the overall ambience of the moment. Of the 20 Heat buttons, only one, the last pressed, was active and the entire lighting rig was treated as one; every look contained “programming” for all the fixtures.
The two columns of buttons to the right of the grid 31 and 33 pertained to audio or MIDI stimulation with the ¾ and Tap buttons aiding the proposed automatic Beats per Second (BPS) detection. With Auto selected, the controller would automatically press a new grid button (chosen randomly) at the start of each musical bar (or specified number of bars) with the BPS determining the rate of any dynamic elements within the look. Strobe, Jog Color and Jog Beam allowed the user to accentuate with strobe effects and to jog the look's color preset and beam settings. The Fever Pitch control 35 was an additional expression device that increased the scale of the dynamic elements of the algorithmic programming (larger pan & tilt movements for example) while the Freeze button 38 would halt all dynamic elements within the look while pressed. The overall concept was to allow a user with no lighting knowledge, such as DJ for example, to busk along to the music, triggering appropriate looks to suit the mood and to provide additional forms of lighting expression.
In more recent times the convergence of video and lighting has opened up further pathways for control which have been enthusiastically adopted by lighting designers. This is the use of media servers as a dynamic source of video data. Such devices may output video signals in many formats which are capable of being used, not only by video display devices such as projectors or video walls, but also by lighting instruments where a pixel or group of pixels of the video image are mapped to individual luminaires. This provides the operator with a level of abstraction that greatly aids the task of dealing with thousands of luminaires. As a single video output from a media server can control the output of many luminaires, changing that single video feed may also change the output of the whole lighting rig. Additionally, some media server manufacturers have developed software and control over their products that allows the operator real time control for live performances over content selection and manipulation of either live video or per-prepared media. The Video Jockey (VJ) systems from companies such as Arkaos are good examples of the sophistication of some of these. However, even these systems require extensive set-up by the operator and are limited in their control, autonomy, and expressiveness.
Appendix A provides an example of how the algorithmic color palettes might be defined. Each set was pre-defined to provide a harmonious mix and that provided the system with a wide range of moods. Appendix B provides examples of how the Heat buttons shown in FIG. 2 might be defined as rules.
In more recent times the convergence of video and lighting has opened up further pathways for control which have been enthusiastically adopted by lighting designers. This is the use of media servers as a dynamic source of video data. Such devices may output video signals in many formats which are capable of being used, not only by video display devices such as projectors or video walls, but also by lighting instruments where a pixel or group of pixels of the video image are mapped to individual luminaires. This provides the operator with a level of abstraction that greatly aids the task of dealing with thousands of luminaires. As a single video output from a media server can control the output of many luminaires, changing that single video feed may also change the output of the whole lighting rig. Additionally, some media server manufacturers have developed software and control over their products that allows the operator real time control for live performances over content selection and manipulation of either live video or per-prepared media. The Video Jockey (VJ) systems from companies such as Arkaos are good examples of the sophistication of some of these. However, even these systems require extensive set-up by the operator and are limited in their control, autonomy, and expressiveness.
If we examine the audio side of the entertainment technology world then we see examples of sophisticated synthesizer systems where a composer or operator can create an entire sound field of voices by modifying root level parameters of a sound signal. This technology dates back to the mid 1950's when Harry Olson & Herbert Belar, both at RCA, completed the world's first electronic synthesizer, the RCA Mk 1. This was followed by the formidable RCA Mk II, funded largely by the Rockefeller Institute, which was acquired and installed at the Columbia-Princetown Electronic Music Centre in 1959. A room-sized, vacuum tube device, the RCA Mk II was programmable via a punched paper roll system, and featured a ground-breaking sequencer. It was complicated and unreliable but hugely influential in that it set out the methodology of subtractive analog synthesis that remains popular to this day. In the early 1960s, Don Buchla & Robert Moog independently developed their own synthesizers that were soon heard throughout the popular music, film and TV scores of the 1960s & 70s. Many other manufacturers followed suit and, today, the synthesizer techniques these early pioneers developed are in use every day in music production and live performance.
A fundamental of these audio synthesizer systems was the use of subtractive analog synthesis where a sound waveform is parameterized down to a few simple but powerful controls that the operator then uses. The general idea was to produce a rich audio waveform using one or more oscillators, then filter out harmonics and finally shape the amplitude, all dynamically and in real time, to create a new and interesting sound. The filtering and amplitude shaping leads to the “subtractive” name even though the first stage, creating multi-timbral waveforms, is really an additive process.
The systems provided an array of building blocks that could be connected together as required. Crucially, every parameter of every module could be modulated by the output of any other module or by dedicated sources. Moog devised the logarithmic (and hence musical) Control Voltage (CV) and Gate scheme which eventually allowed even different manufacturers' modules to work together. Programming these machines came down to connecting modules together with patch cords to route the audio and CV & Gate signals.
The standard modules often included the following functions, in order of the usual signal flow:
Audio:
VCO—Voltage Controlled Oscillator: Outputs an audio waveform such as sine, square, triangle, ramp with the CV setting the frequency of the oscillator. The CV was typically derived from a keyboard.
NG—Noise Generator: A white or pink noise source.
MIXER—Mixer: Combines signals, typically the output of VCOs, noise generators and even external sources. Could also be used to mix CVs.
VCF—Voltage Controlled Filter: Attenuates frequencies/harmonics with the CV perhaps setting the cut-off frequency. Various different responses might be included (low-pass, high-pass, band-pass). CV typically derived from an Envelope Generator (EG).
VCA—Voltage Controlled Amplifier: Varies the amplitude of a signal with the CV typically derived from an Envelope Generator (EG).
Modulation:
EG—Envelope Generator: Triggered by the Gate, generated a CV that followed a user-defined path, typically Attack, Decay, Sustain & Release segments (ADSR), that was then used to shape other parameters. The Gate signal was often derived from a keyboard.
LFO—Low Frequency Oscillator: Like a VCO, but operating at low frequency to generate a varying CV to produce, for example, tremolo (when applied to a VCA) or vibrato (when applied to a VCO).
Keyboard: Generally the primary CV & Gate source.
Pitch bend & mod wheels: Performance controls that added musical expression.
Sequencer: Generated a user-defined, repeating sequence of CVs.
Other modules might include Ring Modulators (combined two audio signals to produce interesting sum/difference harmonics), Sample & Hold and other variants. A critical point in the design of such systems was that any module could be connected to any other module, so the scope for original synthesis was huge. Furthermore, the controls were tactile & immediate, so opportunities for expression and experimentation abounded. This is why, even with powerful digital techniques available, these synthesizers remain popular today.
FIG. 3 illustrates a common arrangement of these audio synthesizer modules and shows the audio, CV 30 and Gate 32 signal paths from module to module. FIG. 3 also illustrates the progression of the audio signal 34 from module to module. The user interface is comprised of the keyboard 40, and mod and pitch wheel 42 and 44 respectively. The system shown shows an LFO 46 serving the pitch 44 and/or Mod 42 wheels. The system shown employ a NG 48 and two VCOs 50 and 52 that are triggered by the keyboard 40. The VCOs and NG send audio signals to a Mixer 54.
The audio signal output by the Mixer 54 is further processed by VCF and VCA modules 56 and 58 respectively supported by modulation provided by respective EGs 60 and 62 respectively.
FIG. 4 illustrates the CV output commonly seen from the ADSR stages of an EG module. For example in FIG. 3 EG2 62 CV output 64. Note that three of the parameters—A (Attack), D (Decay), and R (Release), are times whereas the S (Sustain) parameter is an output level. If an EG module 62 were being driven by a keyboard then the sequence may be as follows.
a. Key is pressed—Output from EG rises 70 over the ‘Attack Time’, A, to an initial maximum.
b. Key is held—Output drops 72 from initial maximum over the ‘Decay Time’, D, to a level 74 defined by the ‘Sustain Level’, S.
c. Key continues to be held—Output remains 76 at ‘Sustain Level’, S.
d. Key is released—Output drops 78 back to zero over the ‘Release Time’, R.
As well as audio synthesizers, we also find video synthesizers to be commonly used in video and television production. These initially followed a similar strategy to audio synthesizers in that the operator controls multiple, low level, inputs which taken together combine to produce a complex output. Video synthesis is a different process to CGI (computer generated imagery) and has become the preserve of video artists rather than television or video production companies and the development has culminated in performance tools such as the GrandVJ from Arkaos.
None of these synthesis techniques have been applied to lighting control in a manner that would allow the combination of mood control and algorithmic programming within the constraints of automated lighting and pixel mapped video. Thus there is a need to expand and improve on the ideas and concepts used in both audio and video synthesizers and to apply them to be used in a system for controlling lighting and video. In particular relating to synthesizing a dynamic lighting configuration in a live environment in response to user input and environmental conditions.